Skip to content

Tool Call Accuracy OpenAPI Tools #42494

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Aug 13, 2025
Merged

Tool Call Accuracy OpenAPI Tools #42494

merged 5 commits into from
Aug 13, 2025
+149 −17

Conversation

singankit
Copy link
Contributor

Description

Please add an informative description that covers that changes made by the pull request and link all relevant issues.

If an SDK is being regenerated based on a new API spec, a link to the pull request containing these API spec changes should be included above.

All SDK Contribution checklist:

  • The pull request does not introduce [breaking changes]
  • CHANGELOG is updated for new features, bug fixes or other significant changes.
  • I have read the contribution guidelines.

General Guidelines and Best Practices

  • Title of the pull request is clear and informative.
  • There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

  • Pull request includes test coverage for the included changes.

@Copilot Copilot AI review requested due to automatic review settings August 13, 2025 00:30
@singankit singankit requested a review from a team as a code owner August 13, 2025 00:30
@github-actions github-actions bot added the Evaluation Issues related to the client library for Azure AI Evaluation label Aug 13, 2025
Copilot
Copilot AI reviewed Aug 13, 2025
View reviewed changes
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR enhances the Tool Call Accuracy evaluator to support OpenAPI tools in addition to regular function tools. OpenAPI tools contain multiple function definitions within a single tool definition, requiring special handling to properly validate tool calls and expand the tool definitions for evaluation.

  • Adds support for OpenAPI tool definitions with embedded function collections
  • Expands OpenAPI tool definitions to individual functions for validation
  • Updates converters to handle OpenAPI tool extraction and model definitions

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.

File Description
test_tool_call_accuracy_evaluator.py Adds comprehensive test case for OpenAPI tool evaluation with currency lookup example
_tool_call_accuracy.py Implements OpenAPI tool expansion logic using itertools.chain to flatten function collections
_models.py Defines OpenAPIToolDefinition class and updates type annotations for tool definitions
_ai_services.py Adds OpenAPI tool extraction logic from thread runs with proper function mapping

@singankit singankit enabled auto-merge (squash) August 13, 2025 17:51
@singankit singankit merged commit 44e41b9 into main Aug 13, 2025
20 checks passed
@singankit singankit deleted the users/singankit/openapi_tools branch August 13, 2025 20:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

Copilot code review Copilot Copilot left review comments

@nagkumar91 nagkumar91 nagkumar91 approved these changes

Assignees
No one assigned
Labels
Evaluation Issues related to the client library for Azure AI Evaluation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@singankit @nagkumar91